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ABSTRACT 

This paper discusses the design and pedagogy underpinning the 
use of online quiz items in which students are presented with a range of 
questions designed to enhance understanding of complex linguistic 
constructions. It explores the design of different types of quiz questions 
from the perspective of pedagogy and cognitive demand. The particular types 
of questions presented in this online learning setting demonstrate a 
formative approach to assessment, closely integrated with learning processes. 
A matrix of questions is presented using Bloom’s taxonomy showing the type of 
question, pedagogical underpinnings and cognitive skills required. The 
implication of the paper is that automated quiz type questions do not 
necessarily imply a narrow focus on recall, but can be designed to assess a 
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Abstract:This paper discusses the design and pedagogy underpinning the use of online 
quiz items in which students are presented with a range of questions designed to 
enhance understanding of complex linguistic constructions. It explores the design of 
different types of quiz question from the perspective of pedagogy and cognitive 
demand. The particular types of questions presented in this online learning setting 
demonstrate a formative approach to assessment, closely integrated with learning 
processes. A matrix of questions is presented using Bloom’s taxonomy showing the 
type of question, pedagogical underpinnings and cognitive skills required. The 
implication of the paper is that automated quiz type questions do not necessarily imply a 
narrow focus on recall, but can be designed to assess a range of learning processes. 



Theoretical background: CAA 

Educators can be in no doubt of the demands of society for lifelong capable learners who are able to perform 
cognitive, metacognitive and metacognitive tasks and demonstrate competencies such as problem solving, 
critical thinking, questioning, searching for information, making judgments and evaluating information 
(Reeves, 2000; Oliver & McLoughlin, 2001). Figure I represents the changing nature of assessment, showing 
a transition from a focus on testing to a focus on learning and transfer of understanding. The traditional 
approach to assessment was largely a form of objective testing which valued students' capacity to memorize 
facts and then recall them during a test situation. Testing was concerned with measuring a range of cognitive 
skills, though many of these tests relied on quantifiable approaches rather than qualitative displays of skills 
and knowledge. Magone et al (1994) called this the one right answer mentality. 
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Figure 1: Continuum of assessment 

The next form of assessment depicted in the continuum is the measurement of competencies, or what we call 
'sequestered problem solving' (Schwartz et al, 2000). In these contexts students are asked to solve problems is 
isolation and without the resources that are typically available in the real world such as texts, Web -re sources 
and peers. Often these tests of aptitude are single shot, and summative rather than formative. In contrast, 
assessment that supports learning and knowledge transfer provides the basis for future learning, and 
continuing motivation to learn (far right of the continuum in Figure 2). This approach is sometimes called the 
alternative assessment movement, as.it is concerned with authentic performance (Gumming & Maxwell, 

1999). Both testing and measuring competence as forms of assessment have been critiqued as being 
controlling, limiting and contrary to student centered teaching and learning. Morgan and O'Reilly (1999) add 
the following criticisms of traditional assessment practices: 

• a lack of variety and autonomy and student choice; 

• lack of applied work-based and project based learning; 
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• overuse of summative forms of assessment; 

• limited use of peer and self-assessment strategies. 

Other indicators of the need to rethink online assessment have come from Bull & McKenna (2000) who 
argue that "the development and integration of computer-aided assessment has been done in an ad hoc 
manner". In a similar vein, Angelo (1999) maintains that we need a more compelling vision of assessment, 
research-based guidelines for learner-centered assessment, and a new mental model of assessment. 



Online assessment: Are multiple choice items appropriate? 

Computer-based assessment using a range of question types has been shown to have positive effects on 
student learning, as students appreciate the speed of marking and the feedback provided (Thelwall, 2000). 
Multiple choice items can be designed to assess a range of competencies relating to student performance 
rather than concentrate on ‘objective’ questions that text recall of discrete facts (Brown et al, 1999). 
Multiple choice items can be used for both formative and summative assessment, and thus to broaden the 
range of skills that is assessed. This paper describes the use of a system of computerized assessment 
utilising a variety of question types to assess higher order skills. Despite the skepticism surrounding the use 
of multiple choice items as a means to assess the skills of analysis, synthesis and evaluation, carefully 
crafted questions can overcome this limitation (Haladyna, 1997). 



Context: Extending the use of multiple choice items 

The context of the study was a year level university course of study in Linguistics delivered online. 

In this unit students are required to develop and apply skills in analyzing the grammar of a language which 
they have not encountered before. Intended learning outcomes are analytical skills, the capacity to 
synthesise conceptual knowledge and 

It was decided to use the quiz feature of WebCT and to create a database driven item bank of questions to 
assess student learning and provide feedback. The learning objective was for students to develop skills in 
linguistic analysis and synthesis. Students are given a database of 84 sentences \n language X, beginning 
with simple sentences (like ‘The woman is going home’), and increasing in complexity until they involve 
quite complex sentences involving constructions like relatives and passives (as in ‘The woman who cooked 
the meat is sitting down.’) Students are supplied with English translations for each sentence, but they do 
not receive any indication of which words mean what, and which of the meaningful subparts of words 
(morphemes) are represented. Using prior knowledge of linguistics, students have to make inferences about 
words parts and combine these in order to interpret the language. Students were required to satisfy 
assessment requirements by completing two separate activities; the construction of a dictionary of this 
language and the completion of a grammatical description of this language. This required higher order 
thinking skills and the capacity to analyze and synthesize language elements. 

Bloom’s taxonomy was used to categorise questions according to cognitive skills required. See Table 1 for 
the matrix summarizing question types. 



Question type 


Link with learning outcome 


Level in Bloom 's 
taxonomy 


Multiple choice 


Learn about and understand how 
to use the lexicon 


Knowledge, Analysis 


Matching items 


Apply new knowledge 


Comprehension, Analysis 


Short answer 


Synthesise new knowledge in 
order to create new sentences 


Application, Synthesis 



Table 1; Matrix showing question type and cognitive skill required 
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Design of the quiz questions 

The quiz items were constructed with three different question styles, designed to assess a range of learning 
outcomes. 

Multiple Choice Questions were used to test skills in the cognitive domain, focusing on comprehension 
and analysis of the grammatical patterns evident in the data. For example in the sample question below, 
students are asked to select variables true of a word class type. 



Question 5 (3 points) 

Which of the following characteristics are true of the class of nouns in this language? [You need to select all the 
characteristics which are true to score full marks, and there may be more than a single correct answer] 

Nouns can occur; 

1 . as a bare stem (unaffixed root) 

2. with the prefix ’ma?-’ 

3. with the possessive suffixes 

4. with the definite suffix ’-e’ 

5. with the suffix-ka? 



Multiple choice items were extended in several ways; 
multiple variables could be true 
all true variables had to be selected for full marks 

all untrue variables were negatively weighted, so that selection of all variables gave 0%. 

Matching Questions were also used to test the skills, focusing on comprehension and analysis of the 
grammatical patterns evident in the data. For example in the sample question below, students are required 
to demonstrate their understanding of word class characteristics by matching each word class type with a 
sample sentence which only that wordclass could fill. Note that the sample sentences are not necessarily in 
the database, and are untranslated. In WebCT Quiz mode, students see all four matching variables 
randomly listed for each subpart of the question. 



Question 2 

Match each word class category listed on the left below with the Lg X structural frame that could only be filled by a word 
that class. 

noun: u-ita-i -e 



verb : 

adjective: 

preposition: 



-i tau-e 

tau -e pole-i 

lao-ka? marege 



Short Answer Questions were used to test students’ ability to synthesise and apply concepts. Application 
and synthesis, as cognitive objectives, are demonstrated by students’ ability to apply general grammatical 
rules they have learned to the creation of new sentences. In other words, students were required to go 
beyond the database given to them, and make up new sentences in Language X which were fully 
grammatical in terms of the grammatical rules they had already worked out. 

The Short Answer style of question allows students to translate a language item and to enter a response, as 
in this sample question. 



Question 36 

How would you say the following sentence in Language X? 

[Use lower case only, use no punctuation, leave only a single space between words, leave no spaces between 
morphemes within a word]. 

The woman who cooked the meat is hit by her father. 

Answer: 
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The marking of questions of this type is also automatic using the Regular Expression syntax feature built, 
into WebCT’s Short Answer shell. This compares students’ responses against a specified string, but unlike 
the Equals and Contains functions. Regular Expression allows predictable variables (is the first letter 
capitalized or not?, has the students put a double space between two word?) to be catered for. It 
additionally allows, still in automated mode, for a range of less than perfect responses to be awarded partial 
marks, for example: 

100%= [M|m]akunrai/s+tunungengngi/s+jukkue/s+ripeddiriwi/s+ri/s+ambo?na 

50%= [M|m]akunrai/s+tunungengngi/s+jukkue/s+peddiriwi/s+ri/s+ambo?na 

50%= [M|m]akunrai/s+tunungengngi/s+jukkue/s+ripeddiri/s+ri/s+ambo?na 

40%= [M|m]akunrai/s+tunungi/s+jukkue/s+ripeddiriwi/s+ri/s+ambo?na 

20%= [M|m]akunrai/s+tunungengngi/s+jukkue/s+peddiri/s+ri/s+ambo?na 



Task 1: Building the dictionary 

Firstly students had to work out the meanings of each of the words and word parts in the data provided to 
them on language X. This was achieved as a collaborative task whereby students contributed their analyses 
of word meanings to an online dictionary available on the unit homepage. This dictionary begins as an 
empty shell with 4 fields of information for each entry. Students are free to add any text to the ‘headword’ 
and ‘meaning’ fields. The ‘word class’ field offers a fixed menu of choices. The ‘name of contributor’ field 
is generated automatically. The dictionary required the development of a specialized tool which does not 
come as a standard feature of WebCT. This tool enables an online class of students to contribute entries to a 
database, and those entries are viewed on screen as though they were a single document. An extract of the 
dictionary looks like Table 2: 



Word 


Class 


Deflnition 


Person 










buaja 


Noun 


crocodile 


Janice L [edit | delete] 


c@mme 


Verb (tr) 


wash 


Sharon E [edit 


delete] 


de? 


Particle 


negative 


Sharon E [edit 


delete] 


galung 


Noun 


field 


Ruth D [edit | delete] 



Table 2: Dictionary extract 

The unit co-ordinator has the capacity to edit and delete any entry, while each student has the capacity to 
edit and delete only the entries that they themselves have contributed. The class as a whole is responsible 
for the accuracy of all entries, so students must negotiate changes to other students’ contributions through 
discussion on the Bulletin Board. This form of collaborative negotiated learning is highly conducive to co- 
construction of meaning and self-directed learning (Coomey & Stephenson, 2001). 

During the semester there were hundreds of postings to the Bulletin Board renegotiating such changes. A 
sample exchange is included below; 

*Hi Joanne et al! I have been mulling over all the *ng's that are placed in differing amounts on 
various words! When, earlier, I had analysed the verb 'drink\ I had thought of it as 'menung' 
rather than 'menungng'. This is becausefrom the available data, both verbs and nouns that end 
in 'ng' seem to only have it as a single sound. Where further instances of'ng' appear on a word 
they seem to be added as suffixes (e.g. 'tedong' - 'tedongnge). However, I may be entirely 
wrong! What do you all think? 
cheers, Sally. 

•Sally, I think the verb is definitely menung, but all words ending in -ng reduplicate this cluster 
before certain suffixes for instance the suffix -i and the suffix -e. (Anna) 
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•Dear Sally, Anna, Joanne & all, 

I agree with Anna about the verb menung. The 'ng' is reduplication before a suffix, (Eileen) 

•Hi guys. 

Yes I can see now that you’re right. . I’ll change this straight away in the 
lexicon. (Joanne) 

The construction of the dictionary was not formally assessed. Each student was asked to contribute, and a 
cap of 5 entries per student was specified to encourage involvement by all students. Students were highly 
motivated to assemble the dictionary as soon as they could, in order to facilitate their capacity to tackle the 
higher level grammatical analysis. All words were entered within the first 3 weeks of the semester, with 
subsequent tinkering, as negotiated on the Bulletin Board, over several more weeks. The lexicon was 
essentially completed by week 6 of the semester. 

Task 2: Completing the grammar 

After analyzing the grammatical structure of this language, students were required to demonstrate their 
understanding of it by completing an online Quiz consisting of 39 questions. Students had the option of 
submitting their Quiz twice. On the first submission, they received back only a grade, but no indication of 
which answers were right or wrong. This mechanism had several advantages; 

it was reassuring to those students who were apprehensive about the technology, and who felt 
uncomfortable with opening a new tool and submitting a Quiz in a single go. These students took 
comfort from a practice run. 

it gave students some means to check that they were on the right track, but without explicitly 
highlighting their weak spots. Most students reported that they found this spurred them on to do 
better. So that on learning that they scored 72% on their first submission, they were motivated to 
do for some further study before the second submission in order to better their score. 



Results 

Analysis showed that the short answer items were the most demanding as it was these questions that 
required students to transfer their knowledge of the lexicon into more abstract forms of understanding. The 
items also required students to synthesise and apply knowledge to create new lexical items, and there was a 
greater range of responses and grades than for the other items. Without a doubt, students who successfully 
completed task 2, (creating the dictionary) demonstrated the desired learning outcomes of analysis and 
synthesis. In all three types of questions, the automated quiz tool and feedback mechanisms provided 
learning support. The pedagogic rationale for the use of these forms of assessment was to align learning 
objectives with assessment tasks, while providing feedback and performance data to learners. The quiz 
questions provided individual and group feedback, allowed students to check their responses and enter an 
alternative answer, and compare their scores with others. In addition, it fostered self-regulated learning as 
learners would access the tests at a time and place convenient to them. 



Conclusions and further research 

Computer-based assessment may suffer an 'image problem' as some assume it is capable only of summative 
testing using multiple choice tests derived from item-banks. Increasingly, computer-based assessment is 
enabling innovative approaches to formative assessment that close the gap between actual and desired 
levels of performance'. A number of factors can be cited concerning the potential value of online learning 
environments and computer-based assessment. Among these are its potential to cater for individual needs, 
foster self-regulation, increase motivation for learning by giving student more control over when and how 
often they test their own knowledge. Current software development and the interactive capabilities of the 
Web enable the creation of procedural, conceptual, cognitive and collaborative assessment tasks and can 
tap students’ prior knowledge, and assess learning in personally meaningful ways that are flexible and 
learner-centered. 
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